1. Field of the Invention
The present invention is a method and system to provide an automatic measurement of people's responses to visual stimulus, based on their facial expressions.
2. Background of the Invention
The human brain constantly processes sensory input to extract useful information and analyze it to understand and properly react. The expressed response is also a very important channel for revealing the internal state or for communication with other individuals, regardless of whether or not the response is voluntary. Understanding the internal psychological state of a human based on the external manifestation has been a subject of study both for scientific and practical importance.
Especially, the current consumer and market-oriented economy put a great deal of importance on people's opinions or responses to various visual stimuli—products, advertisements, or media. Most of the consumers' exposures to such visual stimuli occur in public places or retail spaces at an immeasurably high number and frequency. The ability to capture such occurrences and take measurements of the responses would provide very valuable information to retailers, marketers, or media content providers. Though it is nearly impossible to accurately determine a person's emotional response without directly asking about it, a person usually reveals some indications about it through information channels such as facial expressions or bodily gestures. It is usually the expression on the face that has high correlation with the emotional response.
Recent developments in computer vision and artificial intelligence technology make it possible to detect and track people's behavior from video sequences for further behavioral analysis. Facial image analysis has especially matured, so that faces can be detected and tracked from video images, and the motion of the facial features can also be estimated. The facial appearance changes due to facial expression can be measured to estimate the internal emotional state of a person. A host of academic studies have attempted to recognize the apparent facial expressions or emotional states of humans based on image analysis. These approaches assume more or less ideal imaging conditions and circumstances. The images are typically taken under staged circumstances so that people are instructed to show facial expressions. The facial image analysis and further recognition of expression or emotion can be carried out based on these assumptions. The proposed invention aims to solve the problem under realistic scenarios, where people show natural behavior toward visual stimuli, such as product display, advertisement media, news, movies, etc. It does not strive to measure the response—the changes in attitude or opinion in relation to the content of the visual stimulus. Furthermore, while each instance of such measurement can be erroneous, an accumulated measurement over time will provide reliable information to assess the overall attractiveness of the visual source.
While it is not entirely possible to estimate the mental state of a person based solely on the apparent changes in behavior, the changes in a person's facial expression due to the visual stimulus carry highly relevant information about a person's response through a series of both mental and physical processes. The mental state will trigger the central nervous system to transmit the information to facial muscles. The facial muscle movement then changes the shape of the facial skin so that permanent facial features change shape, or transient features such as facial wrinkles appear or change. These physical changes will render visual changes so that any capable agent (a person or a computing machine) can perceive them. People's faces can appear in any part of the scene with unknown sizes and poses, and their positions, sizes, or poses change over time. The images are also subject to varied lighting conditions. The proposed invention is designed based on this model of information channel from the internal state to the appearance changes; the steps are laid out to solve each reverse problem.
This invention adopts a series of both well-established and novel approaches for facial image processing and analysis to solve these tasks. Face detection and tracking handle the problem of locating faces and making correspondences between detected faces that belong to the same person. Face localization will normalize the facial geometry so that facial features are aligned to standard positions. Under realistic imaging conditions, the extraction of exact facial feature contours can be noisy and erroneous. This invention introduces a novel approach to extract facial appearance changes due to facial expressions; a collection of image gradient filters are designed that match the shapes of facial features or transient features. A filter that spans the whole size of the feature shape does a more robust job of extracting shapes than do local edge detectors, and will especially help to detect weak and fine contours of the wrinkles (transient features) that may otherwise be missed using traditional methods. The set of filters are applied to the aligned facial images, and the emotion-sensitive features are extracted. These features train a learning machine to find the mapping from the appearance changes to facial muscle actions. In an exemplary embodiment, the 32 Action Units from the well-known Facial Action Coding System (FACS, by Ekman & Friesen) are employed. The recognized facial actions can be translated into six emotion categories: Happiness, Sadness, Surprise, Anger, Disgust, and Fear. These categories are known to reflect more fundamental affective states of the mind: Arousal, Valence, and Stance. This invention assumes that these affective states, if estimated, provide information more directly relevant (than do the six emotion categories) to the recognition of people's attitude toward a visual stimulus. For example, the degree of valence directly reveals the positive or negative attitude toward the visual stimulus. The changes in emotional state will then render a trajectory in the three-dimensional affect space. Another novel feature of the invention is to find a mapping from the emotion trajectories to the final response. The central motivation behind this approach is that, while the emotion trajectory already contains very useful information regarding the response of the person to the visual stimulus, there can be still another level of mental process to make a final judgment, such as purchase, opinion, rating, etc. These are the kind of action that ultimately interest the marketers or content providers, and we refer to such process as ‘response’. The emotional trajectory also needs to be interpreted in the context of the dynamics of the visual stimulus. The mapping from the emotion trajectory to the response can also be estimated by training a learning machine using many samples of video sequence along with the ground-truth response data.
There have been prior attempts for detecting and localizing facial features from facial images for the purpose of further facial image analysis.
U.S. Pat. No. 5,781,650 of Lobo, et al. (hereinafter Lobo) disclosed a method for automatically finding facial images of a human face in a digital image, and classifying the age of the person into an age category. Step 1 of the process is to find facial features of the digital image encompassing the chin, sides of the face, and the virtual top of the head, eyes, mouth and nose of the image. Step 2 is to compute the facial feature ratios of the facial features found in Step 1. Step 3 is to compute a wrinkle analysis of the image. Step 4 is to combine the previous two steps to categorize the age of the facial image. The invention can locate and detect facial images for age classification from digital camera images and computerized generated images.
U.S. Pat. No. 5,852,669 of Eleftheriadis, et al. (hereinafter Eleftheriadis) disclosed a method that responds to a video signal representing a succession of frames to detect at least a region of the object. The method processes the video signal to detect the region of the object characterized by a portion of a closed curve and to generate a plurality of parameters associated with the closed curve.
U.S. Pat. No. 6,219,639 of Bakis, et al. (hereinafter Bakis) disclosed a method for recognizing an individual based on attributes associated with the individual. Bakis combined and synchronized biometric and non-biometric features with one another in order to provide a better accuracy of the recognition.
U.S. Pat. No. 7,058,209 of Chen, et al. (hereinafter Chen) disclosed a digital image processing method that detects facial features in a digital image. Chen's method was to apply a geometric reasoning using iris pixel clusters, a summation of squared difference based on iris pixel clusters, or a summation of squared difference method from the pixels in the image.
U.S. Pat. Appl. Pub. No. 2005/0041867 of Loy, et al. (hereinafter Loy) disclosed a method of automatically detecting the location of a face using a pair of eye locations.
U.S. patent application Ser. No. 12/079,276 of Moon, et al. (hereinafter Moon) disclosed a method and system to provide a face-based automatic gender recognition system that utilizes localized facial features and hairstyles of humans. Given a human face detected from a face detector, it is accurately localized to facilitate the facial/hair feature detection and localization. Facial features are more finely localized using the geometrically distributed learning machines. Then the position, size, and appearance information of the facial features are extracted. The facial feature localization essentially decouples geometric and appearance information about facial features, so that a more explicit comparison can be made at the recognition stage. The hairstyle features that possess useful gender information are also extracted based on the hair region segmented, using the color discriminant analysis and the estimated geometry of the face. The gender-sensitive feature vector, made up from the extracted facial and hairstyle features, is fed to the gender recognition machines that have been trained using the same kind of gender-sensitive feature vectors of gallery images.
In Lobo, the facial feature detection is performed under close range high-resolution frontal face images to extract features for age classification. In Eleftheriadis, the facial feature detection is used for image compression, by employing edge and model-based scheme. In Bakis, the lip contour registration is performed for the purpose of multi-modal speaker recognition or verification. In Chen, eyes are detected and localized in a human face, based on the iris color signature and the cluster analysis of the iris color pixels. In Loy, eye candidates are detected first using geometric model of eye images. Based on the eye candidate locations, the facial region is detected, and other facial regions are detected and verified using geometric reasoning (facial features topology). In Moon, a combination of face localization and facial feature localization, based on training multiple learning machines on a large number of data, is used to extract features for recognizing gender.
In most of the mentioned prior inventions, either high-resolution facial images or good quality color facial images are required to reliably detect facial features. The success of these approaches also depends on successful face detection or initial (mostly eyes) features detection. In the proposed invention, an approach similar to Moon is used; the robust facial localization based on a large number of samples is performed after machine learning-based face detection. The facial features are accurately localized within already roughly localized facial feature windows, again using learning machines trained to localize only each given facial feature. The present method does not require high-resolution images or color information; it works with either gray-level or color images, and it works under various imaging conditions due to the training with a large number of images taken under various imaging conditions.
There have been prior attempts for automatically recognizing facial expression of a person using video images.
U.S. Pat. No. 5,774,591 of Black, et al. (hereinafter Black) disclosed a system that tracks human head and facial features over time by analyzing a sequence of images and recognizes facial expression.
U.S. Pat. No. 6,072,496 of Guenter, et al. (hereinafter Guenter) disclosed a method that captures a 3D model of a face for representing facial expressions. Guenter teaches a 3D mesh and a series of deformations of the mesh that define changes in position of the mesh over time.
U.S. Pat. No. 6,879,709 of Tian, et al. (hereinafter Tian-1) disclosed a system for automatically detecting neutral expressionless faces in digital images and video.
U.S. Pat. Appl. Pub. No. 2007/0265507 of Lemos (hereinafter Lemos) disclosed a system and method for determining visual attention and emotional response.
“Measuring facial expressions by computer image analysis,” Psychophysiology, vol. 36, issue 2, by Barlett, et al. (hereinafter Barlett) disclosed a method for facial expressions recognition that applies computer image analysis to the problem of automatically detecting facial actions in sequences of images. Three approaches were compared: holistic spatial analysis, explicit measurement of features such as wrinkles, and estimation of motion flow fields. The three methods were combined in a hybrid system that classified six upper facial actions with 91% accuracy.
“Recognizing Action Units for Facial Expression Analysis,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 2, by Tian, et al. (hereinafter Tian-2) disclosed an automatic face analysis system to analyze facial expressions based on permanent facial features and transient facial features in a nearly frontal-view face image sequence utilizing Action Units (AUs) of the Facial Action Coding System (FACS).
“Active and dynamic information fusion for facial expression understanding from image sequences,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol 27, Issue 5, by Zhang, et al. (hereinafter Zhang) disclosed a method that uses multisensory information fusion technique with dynamic Bayesian networks (DBN) for modeling the temporal behaviors of facial expressions in image sequences.
“Recognition of facial expressions and measurement of levels of interest from video,” IEEE Transactions on Multimedia, Volume 8, Issue 3, by Yeasin, et al. (hereinafter Yeasin) disclosed a spatio-temporal approach in recognizing six facial expressions from visual data.
In Black, the motions of the facial features due to expression are estimated by computing an explicit parametric model of optical flow. The facial feature motions are translated into mid-level predicates, which in turn are used to determine the expression categories. The proposed invention utilizes emotion-sensitive features that extract feature shape changes implicitly, just to be fed to a learning machine to estimate the facial muscle action. In Guenter, the facial actions are estimated in terms of very involved three-dimensional mesh model by tracking a set of dedicated marker points. The present invention strives to estimate the shape change of the facial features just enough to determine the facial muscle action, without using any artificial markers. Tian-1 only aims to detect emotionless faces, while the present invention tries to estimate the change of expressions in a space representing the whole range of human emotions. In Lemos, mostly eye tracking estimates are used to assess the degree of attention and the location of attention within the visual stimulus. The present invention shares a similar goal of estimating human response in relation to a given visual stimulus, but introduces a different focus on the measurement of whole facial feature shapes to determine the emotional changes to a visual stimulus, with specific technical methods to estimate the facial actions, emotional changes, and finally the response. Barlett aims to estimate upper facial Action Units utilizing the holistic, feature-based, and motion (flow)-based image representation and a neural network based learning of the representation. Tian-2 also estimates parametric models of facial feature shapes, and employs neural networks to learn the mapping to the facial Action Units. The present invention also estimates the facial Action Units in an exemplary embodiment of facial muscle actions, and utilizes learning machine to find a mapping from the image representation to the muscle actions. However, the present invention utilizes emotion-sensitive feature extraction scheme, which is different from Barlett or Tian-2. The present invention also utilizes a novel scheme to localize a face and its facial features, while in Barlett the faces are assumed to be aligned. In Zhang, the dynamic change of facial expressions is recognized by a series of methods starting from IR-based eye detection, and facial feature detection based on the eye detection. The facial Action Units recognition is based on deterministic correspondence. The present invention employs novel combination of the face detection, localization, and facial feature localization. The mapping from the facial features shapes to the facial muscle actions is learned by training on a large number of samples.
There have been prior attempts for automatically measuring the audience response to displayed objects or media.
U.S. Pat. No. 7,113,916 of Hill (hereinafter Hill) disclosed a method of assessing consumer reaction to a marketing stimulus.
U.S. Pat. No. 7,120,880 of Dryer, et al. (hereinafter Dryer) disclosed a system for unobtrusively detecting a subject's level of interest, e.g. attention and arousal level, in media content.
U.S. Pat. No. 7,233,684 of Fedorovskaya, et al. (hereinafter Fedorovskaya-1) disclosed a system for using affective information.
U.S. Pat. Appl. Pub. No. 2003/0156304 of Fedorovskaya, et al. (hereinafter Fedorovskaya-2) disclosed a method for collecting and associating affective information for at least one image in an imaging system.
U.S. Pat. Appl. Pub. No. 2003/0032890 of Hazlett, et al. (hereinafter Hazlett) disclosed a method for measuring emotional and cognitive responses to advertising through facial electromyographic techniques.
U.S. patent application Ser. No. 11/491,411 of Thaler (hereinafter Thaler) disclosed a neural network-based rating system that rates records in a database and determines user preference pattern.
Hill aims to measure consumer reaction to marketing stimulus, whose goal is shared by the present invention. However, Hill lists interviewing and manual video coding as tools for collecting opinions and facial expressions. Dryer proposes a system utilizing a host of measurement modalities, such as facial expression, head gesture, or speech, to assess the level of interest to media contents. Fedorovskaya-1 and Fedorovskaya-2 propose systems measuring affective information based on visual image or physiological signal of a person and associating the affective information with the image and person, respectively. Hill, Dryer, Fedorovskaya-1, and Fedorovskaya-2 all propose overall systems, without introducing a very specific novel technical means to achieve the recognition of the response or affective information. The present invention introduces novel technology to automatically extract relevant information from the raw image data and recognize the internal (mental/emotional) state of a human. Hazlett proposes an emotional response measurement based on electromyographic signal of facial muscles, while the present invention processes common visual signal to make the same kind of measurement. Thaler presents a rating system utilizing neural networks, without specific reference to how the input data to the neural network is generated. The present invention also uses learning machines such as neural networks, but the learning machines are trained to process feature vectors that are extracted from video images following novel and specific procedures.
In summary, the present invention provides fully automatic face localization and facial feature localization approaches, for accurately extracting facial and transient features to estimate facial muscle actions due to emotion changes. It is a key novel feature of the present invention to train a learning machine based on the extracted emotion-sensitive features to estimate the facial muscle action; the emotion-sensitive features are designed to extract image features that are highly correlated with the facial expressions. The present invention shares the goal of estimating human response in relation to a given visual stimulus similar to other rating approaches, but it adopts a unique method to predict the final response based on the continuous emotion trajectory, estimated over the course of the visual stimulus.